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Executive Summary 


This study describes the activities and influence of three different types of research practice 
partnerships (RPPs) that shared a common focus on improving mathematics teaching and learning. 
The three types of RPPs were a networked improvement community (NIC), a design research 
partnership (DRP), and a research alliance (RA). These types differ in the kinds of roles researchers 
and practitioners typically play, as well as in their approaches to research. As such, a descriptive 
study has the potential to improve understanding of the relationship between the design of an 
RPP and its influence on the partner district policies and practices. 


The National Center for Research in Policy and Practice (NCRPP), a knowledge-utilization center 
funded by the Institute of Education Sciences, conducted the study. We employed a comparative 
case study approach informed by a conceptual framework that was grounded in previous studies 
of RPPs. We included three RPPs in our study (one of each type). The findings are derived from 
analyses of 63 observations of partnership meetings, 58 different products, 122 surveys, and 133 
interviews across RPPs. 


Data collection sought to elicit information about each partnership’s goals, activities, and 
influence on district policies and practices. It focused on the work of each RPP between 2016 and 
2018. The research questions were: 


1. Howdid the RPPs differ with respect to their organization and major activities? 

2. What kinds of research products did the RPPs produce? 

3. What influence did the RPPs have on partner districts’ decisions regarding policies, 
programs, and practices? 

4. What influence did the RPPs have on sharing research-based ideas with their district- 
leader partners? 


We assigned pseudonyms to the RPPs that participated in the research. We refer to the NIC as 
Eastern Ridge NIC; the DRP is called Cypress Design; and the RA is referred to as Aspen Alliance. 


Key findings from the study were: 


e Allthree RPPs shared acommon long-term goal of improving student outcomes in 
mathematics, but their short-term aims and strategies differed. 


o Eastern Ridge NIC supported testing new teaching strategies in mathematics using 
methods of improvement science. 


o Cypress Design focused on supporting teacher leaders to adapt a model of 
professional development (created by the research team) to their local context, in 
order to improve instruction. 


o Aspen Alliance aimed to provide feedback to its partner on their recent 
implementation strategies, based on a research study focused on malleable factors 
(professional development, implementation) relate to student outcomes. 


e Thecomposition of each RPP differed. Eastern Ridge NIC encompassed the widest range 
of roles and organizations and included eight districts. By contrast, both Cypress Design 
and Aspen Alliance were each partnered with a single district. Cypress Design included 
teacher leaders in the partnership, but Aspen Alliance only worked with central office 
leaders. 


e Allthree RPPs engaged in research activities that are typical of RAs. In addition, both 
Eastern Ridge NIC and Cypress Design engaged in activities typical of a DRP, such as 
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developing, testing, and iteratively refining an innovation designed to improve teaching 
and learning outcomes. Eastern Ridge NIC engaged in typical activities of improvement 
science, including using Plan-Do-Study-Act cycles. 


e Thethree RPPs produced multiple research products between 2017 and 2020. Academic 
conference presentations were most prevalent for Cypress Design and Aspen Alliance; for 
Eastern Ridge NIC, the most common products shared outside the RPP were research- 
based improvement strategies. 


e Eastern Ridge NIC and Cypress Design each had several products co-authored by 
researchers and practitioners, while all Aspen Alliance products were developed by 
researchers. 


e Educational leaders active in the RPPs reported that all three influenced the design of 
professional development in the district. 


e RPPs influenced other policies and practices in partner districts as well. Eastern Ridge NIC 
was credited with providing a new framework for thinking about mathematics. In addition, 
it helped integrate new practices modeled in the partnership. Cypress Design also 
influenced the district in this way, helping to integrate professional learning strategies into 
district practice. Some Aspen Alliance district leaders reported the same for their 
partnership. 


e Education leaders reported that they consulted their research partners in some key 
decisions. RPPs differed with respect to which decisions they were consulted on, however. 
Eastern Ridge NIC was consulted on decisions related to professional development design, 
scaling up a program, and program resource allocation. Cypress Design was also consulted 
on professional development design, while Aspen Alliance was consulted on resource 
allocation. 


e The degree of synchrony and trust among partners was associated with the level of 
influence on district policies and practices. 


e  Inallthree RPPs, there was evidence of key ideas from the RPP being taken up in district 
interviews. The particular ideas taken up reflected both the focus of the work and, in the 
case of Eastern Ridge NIC, the design of the partnership. In Eastern Ridge NIC, district 
leaders also referenced improvement science methods used in that partnership. 


e There was variation across RPPs in the degree to which big ideas were congruent with 
research partners’ sense of those ideas, as well as in the degree to which leaders attributed 
these big ideas to their partners. 
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Introduction 


This report presents analyses of the organization, activities, and products of three research- 
practice partnerships (RPPs) and their influences on partner districts. RPPs are long-term 
collaborations aimed at educational improvement and transformation through engagement with 
research." In contrast to traditional relationships between researchers and practitioners, RPPs are 
intentionally organized to connect diverse forms of expertise and to ensure that all partners have 
a say in the joint work. 


There is no single model for RPPs. Rather, there are different ways that RPPs organize themselves 
to draw on expertise of partners and conduct their joint work.? Some RPPs engage practice 
partners in design and testing of innovations, and even in collaborative analysis of data.? Other 
RPPs preserve distinctive roles for researchers and practitioners, with practitioners maintaining 
the primary role for selecting, adapting, or designing innovations.’ At present, little is known about 
how the organization of an RPP matters for its outcomes—in particular, the decisions made by 
districts, the ways that district leaders engage with research-based ideas from their partners, or 
the products of research that emerge from the joint work. 


This study used a comparative case study design and drew on data collected from three different 
RPPs, all focused on improving mathematics outcomes for K-8 students.° The partnerships 
differed in their organization and activities, and each aligned with one of three broad types of 
RPPs. The aim of the first, a research alliance, was to identify and study how district schools were 
implementing the district plan for new standards in mathematics and science, and to explore links 
between teachers’ instructional practices and student outcomes. The second RPP was a design 
research partnership, in which the purpose was to adapt and test a model of professional 
development for teachers that could be implemented districtwide and that was intended to 
enhance the rigor and relevance of tasks students encountered in classrooms. The third was a 
networked improvement community, which aimed to improve student outcomes in mathematics by 
changing students’ mindsets related to mathematics. (These three types of RPPs are described in 
greater detail later in the report.) 


This report addresses four major research questions (RQs): 


1. Howdid the RPPs differ with respect to their organization and major activities? 

2. What kinds of products did the RPPs produce? 

3. What influence did the RPPs have on partner districts’ decisions regarding policies, 
programs, and policies? 

4. What influence did the RPPs have on sharing research-based ideas with their district 
leader partners? 


All four questions directly relate to the primary aim of the National Center for Research in Policy 
and Practice (NCRPP), which is to understand strategies for supporting educational leaders’ 
engagements with research. RQ 1 describes how different designs of RPPs are organized in ways 
intended to support research use. RQs 2-4 then attend to the RPPs’ progress on different 
research and practice outcomes. 


In answering each of the questions related to products and influences, we consider how the 
organization of each RPP helps to account for variations observed across partnerships, as well as 
how local conditions that may be unique to a particular RPP might matter. For district leaders 
seeking to partner, answers to these questions can inform their thinking about who would make a 
good partner and how to organize the joint work, depending on the needs of that district. Similarly, 
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they can also inform future research to test strategies for improving the effectiveness of 
partnerships. 


Background 


Aset of key interlocking ideas guided the design of this study. To inform the design, we drew ona 
typology of RPPs based on an earlier study of variation in the organization of such partnerships, 
and we describe that typology below.® Next, we considered the range of research outcomes and 
organizational learning outcomes we might expect to see in an RPP. Finally, describe ideas 
pertaining to culture for research use and organizational conditions that may support an 
organization’s ability to productively engage with a partner. 


Types of RPPs 


It is typical for RPPs to engage in a broad range of activities, given the breadth and ambitions of 
their aims. Different types of partnerships can be distinguished by the kinds of work researchers 
and educators do together and separately. A typology developed by Coburn and colleagues® 
depicts three broad types of partnerships: 


e Research Alliances (RAs) are partnerships between a district and an independent research 
organization focused on investigating questions of policy and practice that are central to 
the district. RAs jointly develop research questions with districts and other youth-serving 
organizations, then they conduct the research and funnel findings back to the district, the 
community, and other stakeholders, with the goal of informing policy and improving 
practice in the district. 


e Design Research Partnerships (DRPs) aim to develop, test, and redesign new policies and 
programs with districts and then study the results of these programs using a range of 
research methodologies. Researchers also study fundamental questions related to student 
learning, teacher learning, and/or organizational change in the context of the innovations. 
Thus, their goal is to impact practice and contribute to research knowledge and theory. 


e Networked Improvement Communities (NICs) are networks of districts that seek to leverage 
practitioners’ diverse experiences in multiple settings to advance understandings about 
what works where, when, and under what conditions. They draw on research techniques 
developed from improvement methods in health care to engage researchers and 
practitioners in rapid iterative cycles of design and redesign. NICs use these cycles to 
develop new approaches or adapt existing research-based practices to local conditions in 
ways that address well-defined problems or issues of concern. 


In the current study, we sampled for paradigmatic cases of each type of RPP in order to maximize 
variation in RPP organization and sets of activities. However, because of recent work that 
suggests some hybridization across types,’ we did not assume that each RPP would perfectly fit 
into one of these three types. Indeed, a partnership may engage in different activities based on the 
goals of their work together. Thus, for example, we were open to the possibility that a partnership 
that met the core characteristics of a DRP might also integrate multiple datasets or perform 
independent analyses of district administrative data, activities typical of an RA. 


Outcomes of Partnerships 


Two broad types of outcomes are the focus of this analysis: the development of research products 
and organizational learning outcomes for participating educational agencies. 
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RPP Products 


Contemporary RPPs generate a wide range of products that are important to accomplishing their 
aims. For some, research in an RPP leads to the production of peer-reviewed journal articles. 
These papers might highlight estimates of policy and program effects,® descriptive patterns,’ or 
findings related to implementation.*© Products from RPP research can also take the form of 
research and practice briefs, webinars, or blogs.** Additionally, ideas from research (e.g., about 
how people learn) can be “baked into” the design of tools, materials, or routines, which can then be 
shared broadly, such as when curriculum materials are developed with research-based teaching 
strategies in mind.’ 


Organizational Learning Outcomes for School Districts 


Here, we focus on organizational learning outcomes for participating school districts in the RPPs. 
Organizational learning is the degree to which a participating educational agency integrates ideas 
from the RPP into routines, policies, and collective knowledge that guide their team or 
organization’s behavior.*® 


Organizational learning occurs when an external partner contributes to changes in district policies 
and routines. We define policy broadly, including formal policies as well as rules, plans, and 
guidelines. If adepartment were to integrate ideas from a partner into a new initiative, we would 
consider this to be learning. Likewise, changes in district routines—that is, the patterned ways 
actors in the central office interact with one another or with schools‘*—also indicate 
organizational learning, as when an RPP helps to inform the design of professional development, 
for example. 


RPPs may also contribute to shifts in collective knowledge. Indeed, one of the most powerful 
consequences of engagement with new ideas and experiences is the “transformation of the 
givens,” or the “redefinition of events, alternatives, and concepts.”* In an RPP, ideas shared by the 
research partner can be carried into practice settings in ways that shift how people think about an 
issue. 


Conditions for Productive Partnering 


Based on past studies of partnerships and research use, there are several conditions that could be 
linked to RPP success. 


Culture of Evidence Use 


One likely condition for RPP success is the presence of a culture of evidence use, particularly 
within the practice organization. This conjecture is grounded in studies of research use, which 
have found that use is more likely when a strong culture of evidence is present, one in which 
research is actively sought as a guide to the design of new policies, programs, and practices.*° 
Studies outside RPPs show that the inclination to acquire research when faced with a problem is 
associated with instrumental and conceptual use of research.’” 


Synchrony 


Synchrony is the degree to which partnership activities are conducted in a timely enough manner 
to influence policy and practice and inform decision making. Difficulties in synchrony have been 
cited by RPP leaders as some of the most difficult challenges to overcome.® 
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Trust 


Trust is a key ingredient in partnerships, and it develops as people have more time to interact, get 
to know what motivates different partners, and work together to accomplish aims that require 
each partner to fulfill commitments they have made to one another.’? Many partnership leaders 
consider the level of trust to be a key indicator of the success of an RPP in building relationships 
necessary to engage in ambitious joint endeavors that involve risk taking on the part of practice 
partners.”° A key aspect of relational trust is also the development of affective bonds that develop 
through sustained, reciprocal interaction.”* Another is taking input and concerns of partners 
seriously, such that partners feel understood and respected by one another.” 
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Study Design 


The current study used a descriptive, comparative case study design. We chose three partnerships 
that had different designs but that were focused on a common goal: improving mathematics 
outcomes for middle school students. As this was a descriptive study, no inferences about the 
RPPs’ efficacy can be made, nor was that the intent. 


Population 


In earlier work, two members of the study team developed a research-based typology that 
characterizes existing RPPs.?° As noted earlier, they identified three distinct types of RPPs that 
were then active in school districts: research alliances (RAs), design research partnerships (DRPs), 
and networked improvement communities (NICs). (See descriptions in Background section for 
distinctions among types of partnerships.) 


We used this typology to construct a sample because each type of partnership is organized 
differently to pursue its common goals.‘ Thus, the three types of partnerships provide a useful 
basis for investigating the relationship between partnership design and engagement with 
research. 


First, the three types vary in the nature and intensity of interaction between researchers and 
district leaders. In RAs, researchers and district leaders typically interact at the start of the 
research process to negotiate the focus of the research, and at the end, when they present 
findings and provide opportunities for discussion. By contrast, DRPs and NICs involve more 
intensive collaboration throughout the entire research, design, and development process. 


Second, the RPP types differ in the kinds of research they tend to do and the ways researchers are 
positioned in relation to practitioners. In RAs, researchers typically maintain independence, 
viewing the role of the researcher as contributing research to inform a district’s problem-solving 
efforts, not engaging in those efforts themselves. By contrast, researchers in both DRPs and NICs 
are involved in designing and iteratively refining solutions with district leaders. Regarding 
research methodologies, RAs and DRPs both conduct research that spans from experimental 
design to descriptive studies and, in the case of DRPs, design-based research.”° By contrast, NICs, 
while drawing on a range of research to inform their work, focus most attention on design and 
developing strategies to support local improvement efforts. 


Third, while most partnerships have developed tools and routines for structuring interactions 
between researchers and practitioners, they likely vary in the degree to which these routines 
bring research more centrally into policy deliberations. NICs also tend to be much more likely to 
have intentional strategies to embed inquiry routines into the settings in which they work.?° 


Sampling Procedure 


Asa first step to selecting RPPs, we created a list of possible partnerships, focusing on those that 
could be identified through funding agencies and organizations that support partnerships.’ 
Funding agencies included the Institute of Education Sciences at the U.S. Department of 
Education (Researcher-Practitioner Partnerships, Continuous Improvement Research in 
Education), the National Science Foundation, the Spencer Foundation, and Lucas Education 
Research. Each of these agencies had explicit language calling for partnership research in its 
funding announcements. In addition, we identified candidate partnerships from two networks, the 
National Network of Education Research-Practice Partnerships and the partnerships presenting 
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at the annual Carnegie Foundation Summit on Improvement in Education. For each, we identified 
partner organizations and a description of the partnership’s main focus of work, eliminating any 
that did not focus on mathematics. This yielded an initial pool of 15 RAs, eight DRPs, and five NICs. 


As a second step, we created a brief survey using a Google Form to obtain more information about 
each candidate partnership in mathematics. The form included questions about when the RPP 
began, whether it was currently focused on mathematics with a district partner, and whether the 
work would be continuing over the study period. For sites that met the criteria of being well 
established (that is, partners had been working together for three or more years) and engaged in 
ongoing work related to mathematics, we had initial discussions with partnership leaders to gauge 
their interest in participation. We made final selections based principally on the similarity of work 
being studied in the partnerships, focusing on work focused squarely on studying implementation 
of new standards in mathematics. 


Sample: Description of Three Partnerships 
Below, we provide an overview of each of the partnerships in the study. 
Eastern Ridge NIC 


This NIC was organized by a county office of education in a large state. The county office serves as 
an educational services hub for its 40+ districts; eight of these districts were part of Eastern Ridge 
NIC, serving nearly 100 schools and over 50,000 students. External partners to these districts 
include an intermediary organization that supports the development and maintenance of NICs 
and a well-known mathematics education researcher. The focus of the project was on improving 
achievement in fifth-grade mathematics using methods of improvement science. Their specific aim 
was to increase the percentage of students proficient on state achievement tests by 30% in three 
years. Funding for the project came from a large national foundation. We focused on two small 
districts that were active within Eastern Ridge NIC. 


Cypress Design 


This DRP partnership was between a university and mid-sized urban school district. The focus of 
the work was on adapting a model of professional learning for mathematics teachers. In 
professional learning sessions, teachers analyzed students’ discussions of purposefully chosen 
mathematics problems, using video-recorded lessons from their own classrooms. The aim was to 
help teachers better support student reasoning and problem solving in mathematics. The model 
had been tested in other districts. The partnership work involved both modification of this model 
and research to examine how local leaders adapted the materials. As in other DRP contexts, tool 
development was also part of the project. The project was funded by a federal funding agency. 


Aspen Alliance 


The focal RA was a long-standing partnership between a university research center and a large 
urban school district. The focus of the current study was a specific project conducted jointly with 
one additional research partner, another center at the same university.7® The aim of this project 
was to identify and study how district teachers and schools were implementing the district’s plan 
for new standards in mathematics and science, and to explore links between instructional 
practices and student outcomes. The study was also focused on how changes in practice were 
associated with district strategies for promoting implementation, particularly professional 
development. Of note and in contrast to the other two RPPs, while this partnership did focus on 
middle school mathematics, the project’s focus was broader in both its grade level focus (K-12) 
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and subject matter focus (mathematics and science). Funding for the project came from a federal 
funding agency. 


Data Collection 


The study team relied on a mixture of observations, product lists, survey data, and interviews to 
conduct the study. All data were collected during the 2016-2017 and 2017-2018 school years. 
Data collection for Aspen Alliance began six months later than the other RPPs, due to the length of 
time required to negotiate access to the site. 


Observations 


To help answer RQ 1 (concerning how the RPPs differed with respect to their organization and 
major activities), the study team conducted observations of meetings among RPP leaders and 
participants, as well as of meetings involving only district leaders and only external partners. A 
study team member present as an observer took detailed jottings—that they subsequently turned 
into field notes—that recorded details of interactions, including parts of dialogue among 
participants.” For each observation, in addition to details of interactions, the observer recorded 
which participants were present and what documents were included. 


Over the course of the study, the study team completed a total of 63 observations of meetings: 26 
in Eastern Ridge NIC, 24 in Cypress Design, and 13 in Aspen Alliance. The kinds of meetings 
observed were different for different RPPs, owing to differences in partnership organization. In 
Eastern Ridge NIC, for example, four observations were of network meetings, where county office 
leaders, district and school leaders, coaches, and teachers shared activities they were trying in 
classrooms and what they were learning from them. No similar type of meeting existed in Cypress 
Design or Aspen Alliance. 


RPP Product Lists 


To answer RQ 2, the study team gathered lists of RPP products from the partner leaders. The team 
asked leaders to provide lists and copies of conference presentations, technical reports, journal 
articles, curricula, professional development materials, improvement tools, and other resources 
produced as part of the project and that were available to people outside the partnership. We also 
requested new grant proposals and awarded grants based on the work, as well as any other 
products shared with others that came from the RPP. 


Survey 


The study team administered the survey in spring of the first study year. There were two 
versions—one for external partners and another for district leaders who were involved in 
mathematics decision making in the district. These data sources were gathered to help answer RQ 
a 


The external partner survey elicited goals and activities of the partnership, as well as items related 
to judgments about the expertise of district partners, resources within their own partnership to 
support the work, roles and responsibilities, and perceived capacity to learn from the partnership. 
It also included a pair of 4-item survey scales that the team piloted regarding trust (a = .90) and 
synchrony (a = .76)—that is, the degree to which partners felt they were able to coordinate their 
activities in time to produce relevant research. Finally, this version of the survey included 
questions about how practice partners were involved in research. 
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The district leader survey elicited questions about external partners’ involvement in decision 
making within the district, their internal capacity to benefit from participating in a partnership, 
and the level of research use within the organization. The research-use items drew on previously 
validated measures of instrumental (a = .93) and conceptual (a =.88) research use. In addition, the 
survey included the same trust and synchrony items included on the external partner survey, to 
elicit information about RPP conditions for supporting research use. They had different 
reliabilities from the external partner survey: for trust, reliability was high (a = .92), but for 
synchrony, it was questionable (a = .62). 


The survey window for each partnership was roughly one month, and the study team followed up 
with non-respondents by both email and telephone to secure their participation. The survey 
response rate differed from partnership to partnership. For Aspen Alliance it was 71%, and for 
Cypress Design it was 72%. Eastern Ridge NIC had a lower response rate (62%), due in part to the 
following factors: (1) a large number of participants were not as directly involved in the 
partnership and therefore had less motivation to participate, and (2) one of the other external 
partners administered its own survey concurrently, adding to response burden. 


A total of 122 district leaders and external partners completed the survey, distributed across 
partnership and respondent type as shown in Table 1 below. 


Table 1. Survey and Interview Respondents 


Surveys Interviews 
Partnership District leaders External partners anaes External 
leaders partners 
Eastern Ridge NIC 35 11 25 19 
Cypress Design 36 9 27 18 
Aspen Alliance 23 8 25 19 


Interviews 


As with the survey, the study team created different protocols for external partners and district 
leaders. Interviews helped answer RQ 3 and RQ 4. 


The external partner protocols focused on the history of the RPP, partners’ understandings of 
mathematics problems in the district, key ideas from research being investigated, key activities of 
the RPP—including how the external partners were supporting district work—conceptions of 
partnering, and perceived benefits and challenges of the partnership. 


The district leader protocols focused on leaders’ understandings of mathematics problems in the 
district, key improvement initiatives in the district related to mathematics, key activities of the 
RPP and their involvement in them, the expertise they saw each partner bringing to the 
partnership, research use, conceptions of partnering, and perceived benefits and challenges of the 
partnership. 


The study team conducted a total of 133 interviews over the course of the study, distributed as 
shown in Table 1 above. 
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Issues of Confidentiality and Anonymity 


In this report, no individual partnerships are named, and all names are pseudonyms. We do not use 
any identifiable language related to the individuals or the partnerships. Our reporting focuses on 
broad themes and patterns that emerged. 


Approach to Analysis 


The study team conducted separate analyses of observation data, product lists, survey data, and 
interview data, as described below. 


Analysis of Observation Data (RQ 1) 


As a first step in analysis, all field notes were first anonymized, and then entered into QSR 
International’s qualitative data analysis software package NVivo 12. Next, the study team created 
a topical index to support search and analysis. For the analysis reported here, we identified 
common activity types and then, using a common memorandum format, generated descriptions 
for what case study team members identified as key “signature activities” that repeated or were 
routine. Each memorandum described who the typical participants were, how often they met, 
what activities were performed, what the responsibilities and roles of participants were, how 
expertise was shared within meetings, and what the role of research was within the activity. 
Descriptions in this report are based on these memoranda. 


Analysis of Product Lists (RQ 2) 


The study team entered the product lists into a spreadsheet and, for each product, identified the 
type of product as well as whether it had practice co-authors. To gain a sense of how the focus of 
research varied across RPPs, we also examined the frequency of terms appearing within titles of 
products. 


Analysis of Survey Data (RQ 3) 


The focus of analysis for this report is on a subset of data pertaining to the RPPs’ reports of the 
influence of external research partners and their activities on district policies, programs, and 
practices. The partnerships also reported on the presence or absence of supporting conditions for 
influence, namely the capacity of the external partners, synchrony, trust, and the presence of a 
culture of evidence. 


After data cleaning, the study team developed descriptive statistics (mean, standard deviation); 
these are reported here. The team did not undertake statistical comparisons when comparing 
partnerships, due to differences in roles and respondents. However, looking across partnerships, 
we analyzed the correlations between conditions of research use and reported influence on 
district policies, programs, and practices. 


Analysis of Interview Data (RQ 3 and 4) 


On average, each interview lasted 45 minutes. All were were audio-recorded and then 
transcribed, anonymized, and uploaded to NVivo 12. The team tagged each transcript with 
descriptors for the data collection time period, participant type (district leader or researcher), and 
RPP, so as to be able to track individuals and partnerships. 


The study team proceeded in an iterative fashion to analyze the interview data. First, the team 
engaged in low-inference coding. The goal of this initial stage of coding was to index interviews by 
key constructs, like “challenges.” Lead researchers created a coding guide with major constructs, 
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definitions, relevant interview questions, examples, and non-examples for each code. The team 
revised this coding guide as members coded transcripts, first together and then independently 
once we reached a shared understanding of codes. Four team members engaged in this initial 
coding. Lead team members periodically checked coding to ensure integrity to the coding guide, 
and the whole team met regularly to discuss any issues that arose. 


In order to address RQ 4, the study team asked research team members in each RPP to identify 
the key ideas they sought to convey during the 2017-2018 study year.*° We cross-referenced this 
list with a review of key partner documents. We selected a subset of the most salient ideas and 
developed keywords or phrases for each to identify when these ideas came up. When possible, we 
shared these key ideas and keywords with the research team leaders as a member check, and the 
team revised them accordingly. 


The team then analyzed interview transcripts for the presence of these key ideas and keywords in 
interviews with more and less active district leaders. Mindful that it is possible for educators to 
incorporate ideas in ways that might differ from the original intent,** we analyzed the degree to 
which the interview excerpts with the keywords were congruent with the research partners’ 
original intentions. 
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Goals, Composition, and Approaches to Research 


In this section, we compare and contrast the three RPPs with respect to the organization of their 
partnerships. We use dimensions of variation among RPPs identified by Farrell and colleagues* to 
organize the analysis, which is based on observations of RPPs, survey data, and interviews with 
leaders. Specifically, we focus on the RPPs’ goals, composition, and approaches to research. 


Key findings 


Goals 


All three RPPs shared a common long-term goal—to improve student outcomes in 
mathematics—although each had different short-term aims and strategies for 
advancing that goal. Eastern Ridge NIC supported testing new mathematics strategies 
at school sites. Cypress Design focused on supporting district leaders and school-based 
teacher leaders to provide professional development at their school sites. Aspen 
Alliance aimed to provide feedback to its partner on their recent implementation 
strategies. 


Interms of composition, Eastern Ridge NIC had the widest range of roles and 
organizations represented, and the partnership included eight districts. Cypress Design 
involved research partners, district leaders, and teacher leaders in one district. Aspen 
Alliance focused their work with district leaders, also in a single district. 


All three partnerships engaged in research and inquiry activities that are typical of RAs. 
Eastern Ridge NIC and Cypress Design engaged in activities typical of DPRs, including 
designing, testing, and adapting improvement strategies, as well as in typical NIC 
activities, including conducting rapid cycles of change and developing practical 


At a high level, the three RPPs shared a common aim in the long-term: improving student 
outcomes in mathematics. However, each RPP was organized around different short-term aims 
(see Table 2 below). 
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Table 2. Percentage of External Research Partners Saying Goal Was “Immediate” or “Short Term” 


Partnership 


Eastern 
Ridge Cypress Aspen 
NIC Design Alliance 

Goal (n =5) (n=9) (n=5) 
Develop a deep understanding of the problem the 100% 33% 80% 
partnership set out to investigate 
Identify a specific strategy for improvement and 60% 73% 20% 
develop approaches for evaluating the strategy 
Improve organizational policies or processes that 30% 11% 40% 
directly impact student outcomes 
Improve student outcomes 20% 11% 40% 
Design and carry out research on policies and outcomes . " ‘ 
intended to address the focal problem aun 20% poe 
Build a data infrastructure for the partnership 80% 44% 40% 
Develop research evidence that meets standards for 0% 56% 30% 
publication in a peer-reviewed journal 
Develop research findings that apply to other 0% 33% 30% 
educational organizations 
a capacity of the organization to use 60% 11% 40% 
Develop the capacity of researchers to work in 20% 67% 60% 
partnership with educational leaders 
Build a foundation for future work together 60% 56% 100% 


Eastern Ridge NIC 


For Eastern Ridge NIC, the concrete “aim statement” was focused on achieving a specific and 
measurable improvement in student test scores. The strategies for achieving the RPP’s goals were 
to change classroom culture and mathematical mindsets of students, support more sensemaking 
and productive struggle within mathematics instruction, align supports for instructional 
improvement at the school, district, and county level, and promote collective learning within 
Eastern Ridge NIC. As in other NICs, this RPP sought to use iterative, small tests of change—trying 
out new strategies and measuring their effects—to make improvements to the quality of 
instruction. 


In the survey of Eastern Ridge NIC’s external partners, there was the greatest consensus around 
the following short-term goals: developing an understanding of the problem (n = 5 of 5, 100%), a 
focus on improving policies and processes that can impact student outcomes (n = 4 of 5, 80%), and 
building a data infrastructure for the partnership (n = 4 of 5, 80%). These are all essential for 
building a strong foundation for an NIC at its initiation.?° 
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Cypress Design 


For Cypress Design, the targeted student learning goals emphasize students’ engagement in 
mathematical practices. The focal project of Cypress Design was a proof-of-concept for the scaling 
capacity of its teacher professional development model to prepare students to engage in such 
practices. These efforts would then, if successful, improve teacher instruction and students’ 
abilities to construct viable mathematical arguments. The approach in this case was to adapt a 
research-based model of a professional development cycle focused on analysis of a cognitively 
demanding mathematics task, followed by video-based analysis of teachers implementing the task. 
In past studies, this model had been led by external researchers, and it had shown positive effects 
on teacher knowledge and instructional practice. 


In their survey reports, Cypress Design’s external research partners had the greatest agreement 
around the short-term goals of developing a specific strategy for improvement and approaches for 
evaluating the strategy (n = 7 of 9, 78%) along with developing the internal capacity of the 
research team to carry out work in a partnership fashion (n = 6 of 9, 67%). The focus on identifying 
strategies for improvement is consistent with the interventionist nature of design-based 
research.** In addition, a focus on internal capacity building is consistent with observations among 
design-based researchers that there is a strong need to develop capabilities for long-term 
partnerships with districts. 


Aspen Alliance 


Finally, Aspen Alliance’s project focused on studying the district’s plan for implementing the 
Common Core State Standards in Mathematics and the Next Generation Science Standards. The 
goal of the research was to understand the relationships among centralized implementation plans, 
professional development participation, malleable factors at the school level, instructional quality, 
and student outcomes. In the long run, the partnership was interested in supporting improvement 
on student achievement in mathematics and science, as measured on standardized tests, grades, 
and student self-reports. 


The short-term goals identified by Aspen Alliance researchers were focused on building a 
foundation for future work together (n = 5 of 5, 100%). While the broader RPP had an extensive 
history, this specific project was in its early days, and it involved new district leaders. This goal was 
followed by a focus on developing a greater empirical understanding of the focal problem (n = 4 of 
5, 80%), designing and carrying out research on policies and outcomes intended to address the 
focal problem (n = 4 of 5, 80%), and developing research evidence that meets the standard for 
publication (n = 4 of 5, 80%) or could apply to other educational settings (n = 4 of 5,80%). While 
RAs typically are mainly focused on developing findings useful to their own partner district, and 
not necessarily beyond it, it may be that this RA’s past success and influence on other districts’ 
work made this goal more of a priority. 


One notable cross-case pattern is that none of the research partners said that improving student 
outcomes was the most important short-term priority. A closer look across partnerships, however, 
reveals that all but one researcher (n = 18 of 19, 95%) said improving student outcomes was a goal 
the RPP hoped to achieve either 1-2 years from now or, for 80% of respondents, over the long 
term. In other words, they believed it was a goal that would require a longer time horizon to 
accomplish. 
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Composition 


Partnership composition varied across the three RPPs. Who was involved in each partnership was 
aligned with RPP type, although it reflected the specific goals of the partnership as well as the 
location and size of the district (or districts) involved. 


Eastern Ridge NIC 


By far, Eastern Ridge NIC had the widest range of roles and organizations represented among 
active RPP participants. The active participants included teachers, coaches, principals, district 
leaders, and county office of education leaders, who served as the “hub” for the network and who 
were responsible for organizing all of the meetings. There were two different external research 
partners, each of whom played a distinctive role. One was an organization that played what it 
called an “analytic support” role within the RPP, assisting the county office leaders with learning 
how to implement the tools of improvement science, from Plan-Do-Study-Act cycles (see p. 22) to 
practical measurement strategies** for assessing change in teaching practice. A second research 
partner from a university in the region brought substantive expertise in mathematics education, 
along with a set of resources to be adapted for the NIC context. 


Cypress Design 


Cypress Design was similarly a multitiered partnership,?” meaning people at different levels of the 
district were represented in core activities—from district mathematics curriculum office leaders 
to building-level instructional support staff (e.g., coaches). In addition, the partnership leadership 
team comprised a research team from a local university—which brought expertise in mathematics 
education, teacher learning, and design-based research methods—and an organization that served 
as a partnership broker. 


Aspen Alliance 


Aspen Alliance was made up of a small team of curriculum leaders at the district office and 
representatives from two research organizations, both housed at the same university. One of the 
organizations was a long-standing RA partner to the district; the other specialized in the study of 
implementation of STEM curricula and programs. An additional organization, not formally part of 
the partnership for the project, was integral to designing and implementing the professional 
development activities offered to teachers as part of the district’s improvement strategy. 


Approaches to Research 


From the original typology we identified different kinds of research activities in which RPPs 
engage. We were interested in the degree to which our case RPPs engaged only in activities 
typical of their partnership type, or if we might see evidence of hybridization, as reported 
elsewhere.°? 


Table 3 below summarizes by partnership the kinds of research activities undertaken by each RPP, 
as reported by the researchers in that partnership. We classified a partnership as engaging in this 
activity if aresearcher reported so in the survey. 
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Table 3. Different Forms of Research Activities Pursued by RPPs, As Named by Research Partners 


Partnership 
Eastern Cypres Aspen 


Ridge S Allianc 
NIC Design e 

Activity (n= 5) (n= 9) (n= 5) 
RA-like activities 

Build a data archive or database that combines 

multiple datasets v a v 

Analyze the implementation of a policy or program in 

the organization v we vi 

Analyze the outcomes of a policy or program in the 

organization v xC v 

Examine associations between malleable factors (e.g., 

program implementation) and outcomes of v 

Identify gaps in student outcomes Vv v Vv 

Interpret data together Vv Vv v 

Communicate results of data analysis to others in the e. e. 

organization 
DRP-like activities 

Develop a set of conjectures about features of 

programs needed to address a problem for the Vv Vv Vv 

organization 

Jointly design a program or intervention that 

addresses a problem in the organization 

Try out a program or intervention that the 

f ; Vv Vv 
partnership designed 
Adapt or refine a program the partnership designed, VY y 


based on testing it in the organization 

NIC-like activities 
Use tools from improvement science (e.g., root cause 
analysis) to represent understanding of a problem in Vv 
the organization 
Conduct rapid tests of change (e.g., a Plan-Do-Study- 


Act cycle) to address a problem in the organization - ~ 
Develop “practical measures” to test whether a 

change is an improvement - “ 
Create a network of people interested in testing y , 


solutions to a problem in the organization 


We found that all three partnerships engaged in activities that are typical of RAs, including 
building a data archive, analyzing implementation and outcomes results, and looking at data 
together. Further, two of the three partnerships—Eastern Ridge NIC and Cypress Design— 
engaged in activities typical of DRPs, including designing, testing, and adapting improvement 
strategies. Both of these RPPs also reported engaging in typical NIC activities, including 
conducting rapid cycles of change and developing practical measures. 
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Overall, Aspen Alliance was the most “true to type” of the three partnerships. The one exception 
for this RA was the DRP strategy related to developing conjectures about how district strategies 
could result in improvements in student outcomes. Not only was such a model included in the 
project proposal, but it is fairly common for RAs’ work to be guided by a theory of action. 


The boundaries were more blurred with Cypress Design and Eastern Ridge NIC. The DRP and NIC 
approaches share a commitment to co-design and testing of innovation that distinguishes them 
from RAs. It has also been said they belong to a common “family” of approaches to educational 
improvement.?? 


Signature Activities of RPPs 


The survey results above help us understand the range of research approaches taken in these 
RPPs, but we are left with questions about how these activities fit together to advance the RPPs’ 
specific goals. Next, we offer in-depth descriptions of how each RPP organized its research 
activities to advance its aim. For each, we highlight a signature activity that we found essential to 
that RPP’s work. 


Eastern Ridge NIC 


In Eastern Ridge NIC, the primary approach to inquiry was to engage in systematic inquiry into 
change strategies in mathematics using methods of improvement science, as adapted from 
medicine and other fields to an educational context.*° These methods include the use of tools, such 
as aim statements, to set specific, measurable targets for improvement; use of “driver diagrams” to 
represent a working theory of improvement; development of asystem of practical measures for 
studying short-term improvements to practice; and use of a structured process for testing 
changes called Plan-Do-Study-Act cycles.*? The inset below describes how, during Eastern Ridge 
NIC meetings, the Plan-Do-Study-Act Cycles were developed. 
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Eastern Ridge NIC Signature Activity: Network Meetings 


A key idea behind an NIC is that network members should have opportunities to learn from each other 
by sharing what they are learning from testing interventions in the classroom.” Teams use a common 
improvement methodology to guide their local learning, in this case the Model for Improvement 
outlined by Bryk and colleagues. In Eastern Ridge NIC, network meetings provided opportunities for 
network members to share projects, learn about research based practices, and plan the next action 
period. One of the tools for structuring learning is an improvement cycle called Plan-Do-Study-Act 
(PDSA): Look together at data, get training in improvement methods, reflect on the previous action 
period, and plan for the next action period, which includes doing the next PDSA. In the NIC, different 
districts may be conducting multiple PDSA cycles simultaneously, to learn quickly strategies that show 
promise. These meetings took place once a quarter and, along with twice-yearly visits to other districts, 
were one of the only opportunities for cross-district collaboration and networking. They involved teams 
of teachers and administrators from all eight districts in the network, as well as county office of 
education leaders. The county office leaders worked together to plan, coordinate, and facilitate network 
meetings, though there have been routine opportunities for district-level teams to organize their own 
small-group discussions and choices about what to share with the larger network. 


Each meeting started with an ice breaker intended to promote a culture of trust and social connections 
between network members. This activity was followed by an opportunity for participants to engage ina 
mathematics problem together (i.e., solve it as if they were students) to build their comfort and 
competency with mathematics. There was built-in time for sharing learning from past cycles, where 
network members could learn about a new change idea to try from colleagues in other districts. There 
was always time, too, for analyzing data together. Network members worked in teams to look at cross- 
district data from common assessments that teachers give to their students. Members also shared 
information and learning from previous PDSA cycles, including the strategies they were trying, what they 
had discovered, what worked and did not work, and what they were going to do differently next time. 
Finally, there was time to plan the improvement work for the next cycle, where district teams decided 
ona change to test in their own context. Informing this planning were templates for a series of PDSA 
cycles and research-based ideas from the mathematics researcher, as well as ideas that district teams 
may have heard from other teams that inspired them. 


In network meetings, research was present as a method for systematically investigating interventions to 
improve practice, but practitioners’ voices were in a lead role. Teachers’ expertise, in particular, was 
highly valued, particularly in the planning of the PDSAs, because they were able to decide what change 
ideas they wanted to test in the classrooms. Consistent with the purpose of a NIC, the goal was to 
produce reliable practice-based evidence by engaging in PDSA cycles and collective sensemaking across 
the network. 


In order to make progress on the aim of improving fifth-grade mathematics proficiency, Eastern 
Ridge NIC’s working theory of improvement emphasized the need to change classroom culture 
and mathematical mindsets of students, support more sensemaking and productive struggle 
within mathematics instruction, align supports for instructional improvement at the school, 
district, and county level, and promote collective learning within Eastern Ridge NIC. To measure 
the success of these strategies, this RPP relied on a mixture of longer-term measures (e.g., 
proficiency on state tests) and short-term, practical measures, including a student exit slip anda 
place for teachers to log implementation of practices. Exit-slip questions were typical of practical 
measures that are easy to implement and close to outcomes that NICs seek to improve. Examples 
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include, “Today | learned how my ideas connect to other ideas” and “Today | learned about how 
other students think about math,” which are both keyed to the model of mathematical 
sensemaking promoted within Eastern Ridge NIC. 


In addition, the external researcher for the project conducted a quasi-experimental study that 
compared student outcomes for teachers who received professional development with student 
outcomes for teachers who did not. This type of study is not common within NICs, though there 
are some examples.” 


Cypress Design 


In Cypress Design, the primary approach to inquiry was design-based implementation research 
(DBIR).“ DBIR adheres to four main principles: (1) Teams form around a focus on shared goals that 
address persistent problems of educational inequity identified through negotiation among 
multiple stakeholders’ perspectives and values; (2) to improve practice, teams commit to iterative, 
collaborative design; (3) to promote quality in the research and development process, teams 
develop theory, knowledge, and practical tools related to both learning and implementation 
through systematic inquiry; and (4) DBIR is concerned with developing capacity for sustaining 
change in systems.*° 


The Cypress Design focal project reflects these principles in terms of how inquiry was organized. 
The teams developed shared goals for improvement through a collaborative proposal 
development process. Though they did not co-design the professional development from scratch, 
they co-adapted it, integrating key elements in a principled way, with district curricular resources 
and infrastructures for instructional support. The research team created a range of practical tools 
for implementing professional development, such as protocols for facilitators to use to rehearse 
professional development activities. They also developed research protocols for documenting 
professional development and studying teachers’ implementation of tasks in classrooms, adapting 
research protocols used in the research team’s previous studies. And, the team sought to develop 
the capacity of district instructional support leaders to provide the professional development for 
teacher leaders on their own, over time. The inset below describes a signature activity within 
Cypress Design that shows what typical co-adaptation meetings with education leaders looked 
like. 


Cypress Design Signature Activity: Adapting the Professional Development Model 


In quarterly meetings, district leaders and researchers from the partner university engaged in the work 
of adapting the researcher-developed professional development model, given the partner district’s 
priorities and constraints. They were scheduled to fit within the time frame for professional 
development implementation, as outlined in the grant. Agendas for the meetings were co-constructed 
by district leaders and researchers ahead of the meeting, and these agendas then structured their time 
together. Some meetings included just the core leaders of the RPP, but others included a wider planning 
team that encompassed research assistants and mathematics coaches in the district. When more people 
were present, time was often dedicated to reviewing norms for participation, a common practice within 
the district. One of those norms included attention to valuing many perspectives and voices in the room. 
Again, when more people were present, there was also typically time for looking at data together from 
past workshops and for making sense of patterns of the data. There was small-group work and 
discussions where all people were expected to share their perspectives. One of the university-based 
researchers typically facilitated the meetings, which involved overseeing meeting logistics, opening the 
meetings, presenting the agenda, and keeping the group on track in following the agenda. Facilitation 
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was active and often involved calling on particular individuals to share their perspectives; this ensured 
that relevant expertise was brought to bear in discussion and that all voices were heard. 


Aspen Alliance 


Aspen Alliance’s project focused on malleable factors associated with changes in student 
outcomes. Those factors focused on school- and district-level conditions as well as teacher 
professional development. In addition to modeling student outcomes, the research team modeled 
implementation of standards-aligned resources and practices, drawing on interviews with 
teachers and school leaders as well as a set of measures they developed to complement others 
taken from their regular district surveys of students and teachers. Important features of this RPP 
were its multiyear data archive that could be drawn upon for analysis and its annual surveys that 
could be adjusted to accommodate specific study questions. 


The creation and use of data archives are common among RAs. Such archives typically allow for 
linkage of administrative datasets with data collected by researchers for specific studies. Aspen 
Alliance projects have sometimes also used mixed-methods approaches; most often, they have 
emphasized quantitative modeling of outcomes, using qualitative research to help interpret 
results of those models. 


Aspen Alliance Signature Activity: Feedback Sessions 


Aspen Alliance feedback sessions were quarterly meetings between the Aspen research team and its 
district contacts. In these sessions, the research team shared updates on progress in data collection and 
analysis and asked for feedback from their district partners. These were scheduled for times when the 
research team anticipated needing input from the district or wanted to provide updates. The research 
team sometimes requested feedback on proposed instruments, the value of analyses performed, or 
upcoming data collection plans. They may also have had specific questions for the district leaders or 
specific requests—for example, help with identifying sites for data collection. Typically in attendance 
were the principal investigators and at least one district representative, the head of curriculum for 
mathematics and science. The research team was responsible for preparing the agenda, and usually 
came with a slide presentation to share with the district leaders. In this sense, researchers played a lead 
role, and district expertise was elicited primarily to interpret and make sense of findings and assist with 
future data collection. It was an opportunity for both researchers and leaders to learn: Leaders learned 
about key emerging findings, while researchers deepened their knowledge of developments in the 
district policy context that could inform their interpretation of findings. 
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Research Products of RPPs 


A key aim of RPPs is to contribute to knowledge- and theory-building beyond the partnership 
itself.*° In addition to peer-reviewed journal articles, RPPs produce technical reports, give 
conference presentations to academic, professional, and hybrid audiences, and develop 
educational resources that can be shared with others. In RPPs, practice partners may act as 
authors, co-authors, and co-presenters. A comparative analysis of research products, then, needs 
to consider the breadth of products, audiences, focus, and authorship. 


Breadth and Audiences of Research Products 


The most common venue where RPPs shared their work was academic conferences, followed by 
the “other product” category and peer-reviewed journal articles. All shared their research at one 
or more conferences where practitioners were present (see Table 4 below). 


Table 4. Research Product Outlets, 2017-2020 


Partnership 


Eastern Ridge . } 
Outlet NIC Cypress Design Aspen Alliance 
Peer-reviewed journal ‘ ; : 
articles 
Conference presentations 
Academic 3 21 5 
Practitioner 1 1 0 
Hybrid 
(academic/practice) 1 1 al 
Other products 10 1 A 
Total 16 % 11 


A closer look, however, reveals some differences among the RPPs. Overall, Cypress Design shared 
more products outside the RPP than the other two partnerships (n = 26). Cypress Design had the 
highest number of academic conference presentations (n = 21), made principally to subgroups of 
researchers focused on mathematics learning. Eastern Ridge NIC produced 16 different products, 


Technical Report No. 4 « 26 
National Center for Research in Policy and Practice 


the largest number of which were classified as “other,” because they were intervention packages 
shared externally. Finally, Aspen Alliance principally focused on academic conference papers (n = 
5) and other kinds of products, largely analytic reports (n = 4). 


Differences among partnerships in types and numbers of products could reflect different 
emphases within them on communicating with different audiences. NICs, for example, prioritize 
the creation of bundles of resources called “improvement packages” that are developed, tested, 
and refined using evidence collected by teachers implementing them.*” Many DRPs participate in 
the academic community of learning scientists; they also write for practice audiences, but papers 
can take some time to develop as researchers engage in retrospective analysis.*® And RAs typically 
produce technical reports that are made public. 


Focus of Research Products 


Next, we analyzed the content of the research products generated by the RPPs. An examination of 
the most commonly appearing words in the titles shows some differences among the partnerships 
(Table 5). While mathematics appeared as a term in all three partnerships, it was in the top seven 
words that appeared only for Eastern Ridge NIC and Aspen Alliance. For Cypress Design, terms 
related to qualities of mathematics instruction, such as “discussion,” were common. Also, it is 
possible that the term mathematics was not at the top of the list because many presentations were 
at mathematics education conferences, and thus presumed a mathematics focus. In addition, 
Cypress Design had a focus on professional development and attributes of professional 
development being designed and studied in the partnership (e.g., video-based). 


Table 5. Commonly Appearing Terms in RPP Research Products 


Eastern Ridge NIC Cypress Design Aspen Alliance 
Network/ed Video-based Standards 
Learning Teacher/s Mathematics 
Mathematics Research Science 
Improvement Professional Common Core 
Mindset Development Instruction 
Social Partnership Achievement 
Analysis Discussions Implementation 


The influence of each partnership’s design can be seen in the focus of its research products. 
Eastern Ridge NIC’s research publications focus specifically on the network and sharing within it; 
this is tied to the basic assumption that an NIC thrives through knowledge sharing among nodes in 
the network. Interestingly, the focus on the network itself appears to be more prominent than the 
focus on mathematics. In DRPs, a focus on subject-matter learning is key. Specific terms that refer 
to desired characteristics of instruction—like “authority,” which refers to giving students 
intellectual authority in developing ideas in the classroom—are common in titles, which reflects 
the connection with district priorities, since “agency,” “authority,” and “identity” are keywords in 
district mathematics guidance documents. Aspen Alliance’s products similarly focus on 
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mathematics, but those specific instructional terms do not appear. Here, standards, achievement, 
and implementation are key terms. This is not surprising, given Aspen Alliance’s focus on studying 
implementation and links between implementation and student outcomes. 


Authorship 


Finally, we analyzed authorship to determine if each research product was developed by 
researchers alone, practitioners alone, or a mixed authorship team with both researchers and 
practitioners. 


Two of the three partnerships included practitioners as co-authors on their products. Eastern 
Ridge NIC had two products with practitioner co-authors; both were conference presentations 
aimed at mixed audiences of practitioners and researchers. As everyone within an NIC is seen as 
an investigator, it is not surprising that the percentage of co-authored products (25%) is high 
relative to the other two RPPs (12% for Cypress, 0% for Aspen). Cypress Design had three 
products with practitioner co-authors, one at a research conference, another at a practitioner 
conference, and a third that was a multimedia product describing the project’s activities. Finally, 
for Aspen Alliance, the absence of co-authors from the practice organization is to be expected. In 
an RA, researchers and practitioners often have distinct roles; it is researchers who conduct the 
research and present results, and practitioners who take responsibility for selecting, adapting, or 
designing solutions to focal problems. 
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Influence of RPPs on Partner Districts’ Policies, Practices, Routines, and Decision 
Making 


Some expect that RPPs are well positioned to support organizational learning in partnering 
educational agencies. One dimension of organizational learning is the degree to which aresearch 
partner influences the participating educational agency’s policies, practices, and routines. In this 
section, we answer the question of whether and how the three RPPs influenced these three 
aspects of district functioning. We analyze, too, when and for what types of decisions external 
research partners were consulted by district partners. 


Key findings 

e Educational leaders active in all three RPPs reported that their partnerships influenced 
the design of professional development. The majority of leaders in Eastern Ridge NIC 
reported that the RPP had provided a new framework for thinking and helped support 
the integration of new practices modeled in the partnership. The majority of leaders in 
Cypress Design reported that the RPP had supported the design and integration of new 
practices into the district. And about a third of surveyed leaders in Aspen Alliance 
reported integrating new practices from the RPP into the district. 


e Educational leaders reported that they consulted their research partners in key 
decisions. Eastern Ridge NIC partners were often consulted in decisions related to 
professional development design, scaling up a program, and resource allocation. Less 
frequently, Cypress Design district leaders who were involved in redesigning a program 
or designing professional development consulted their partners, while in Aspen 
Alliance, the district leaders in involved in resource allocation decisions consulted their 
partners. 


e Twoconditions—synchrony and trust—predicted RPP influence on policy practices and 
routines. 


Influences on Policies, Practices, and Routines 


Many partnerships seek to impact local policies and practice. Research from a partnership can 
inform a district’s search for better organizational policies, processes, and structures, as well as 
productive strategies for improving leadership and teaching practices.*® Ultimately, 
implementation of these changes is intended to result in improved student outcomes. 


Our survey asked district leaders what kinds of district-level policies and practices external 
research partners had contributed to (Table 6). Across all three partnerships, district leaders 
active in the RPP reported that external partners most frequently influenced professional 
development (n = 16 of 21, 76% for Eastern Ridge NIC; n = 8 of 10, 80% for Cypress Design; n= 9 
of 14, 64% for Aspen Alliance). For Eastern Ridge NIC, leaders reported that the RPP had provided 
anew framework for thinking (n = 16 of 21, 76%) and helped support the integration of new 
practices modeled in the partnership (n = 15 of 21, 71%). Seven of the 10 active leaders involved in 
Cypress Design (70%) reported that the RPP had supported the design and integration of new 
practices into the district. 
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Table 6. Ways Educational Leaders Said External Partners Helped Shaped District Policy and 
Practice 


Partnership 


Eastern Ridge Cypress Assemallianbe 
NIC Design (n= 14) 
Type of influence (n = 21) (n = 10) 
pew or revised policy in the 10% 20% 21% 
istrict 
A decision to adopt a new 
program or intervention in the 19% 0% 0% 
district 
The design of a new practice or 
program to address the goals of 52% 70% 7% 


the partnership 


The design of professional 
development related to the goals 76% 80% 64% 
of the partnership 


Anew framework or set of ideas 
to help think about the goals and 76% 30% 21% 
how to address the problem(s) 


The integration of new practices 
in my district that were 
developed or modeled in the 
partnership 

Note: For this table, sample includes only leaders who said they were active in the RPP. 


71% 70% 29% 


Research Partner Consultation in District Decisions 


Direct involvement of research partners in decision making is another possible mechanism of 
influence on district policies and practices. As Table 7 shows, educational leaders did indeed 
consult their RPP external partners in making a range of decisions related to mathematics. More 
than half of the district leaders involved in such decisions reported that the external partners of 
Eastern Ridge NIC were consulted on professional development design, scaling up a program, and 
resource allocation. In Cypress Design, about one quarter of the district leaders involved in 
redesigning a program or designing professional development consulted their partners. And in 
Aspen Alliance, about one fifth of the district leaders involved in resource allocation decisions 
consulted their external partners. 
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Table 7. Ways External Partners Were Consulted in District Decisions 


Partnership 


Eastern Cypress Aspen 
Type of consultation Ridge NIC Design Alliance 
Conducted a major adoption of curriculum 21% 7% 0% 
materials (n= 14*) (n= 14) (n= 14) 
Considered purchasing a particular 4 . . 
intervention or program targeted at a specific ne meen mar 
student population 
F , i 57% 14% 14% 
Considered scaling up a pilot program (n= 14) (n= 14) (n= 14) 
. 23% 25% 0% 
Redesigned a program (n = 13) (n= 12) (n= 14) 
. . 62% 24% 14% 
Designed professional development (n = 21) (n= 29) (n= 14) 
F eee . 25% 0% 14% 
Considered eliminating a program or policy (n = 12) (n= 11) (n= 14) 
Considered directing new or additional 579% 11% 21% 
0 (0) (0) 


resources (funds and/or people) to a particular (n= 14) (n= 18) (n= 14) 
program 


Note: Respondents only answered the questions pertaining to decisions they were involved in. As such, the 
sample size varies by item. 


The overall larger influence of external partners on professional development practice and 
decisions is likely related to the focus of the projects we studied, since all involved implementation 
of some form of mathematics-related professional development. At the same time, this influence is 
notable: designing professional development is one of the most common activities in which school 
and district leaders make decisions. It suggests that RPPs of different types could be mechanisms 
for facilitating research use ina common activity within school districts. 


We also suspect that these findings reflect the different orientations RPPs have regarding roles 
and the degree to which research partners should be involved in educational decisions. In this 
regard, the RA model has a more traditional stance than the DRP and NIC models. That is, in an RA 
model, external partners are involved in conducting research and producing findings, but it rests 
on the educational leaders to determine the implications of those findings. 


Conditions Associated with RPP Influence on Districts’ Policies, Practices, and Routines 


District leaders in all three RPPs used research for instrumental and conceptual purposes at levels 
comparable to those in a nationally representative sample of leaders.°° Specifically, they used 
research frequently to inform decisions (instrumental use) and influence their thinking on issues 
(conceptual use). The levels of research use were highest among district leaders within Aspen 
Alliance. 


Instrumental use of research. Table 8 below shows how frequently, on average, leaders in each 
partner district used research for different instrumental reasons. The sample size varies by survey 
item, since the responsibility for each specific decision was not shared by all respondents in the 
survey. 
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Table 8. Instrumental Research Use Among District Leaders 


Partnership 


Eastern 
Ridge NIC Cypress Aspen 
(12<n< Design Alliance 
Use of research 29) (1411<n<30) (6<n<22) 
Conducted a major adoption of M = 1.86 M=2.21 M = 2.36 
curriculum materials SD =.95 SD = .80 SD = .81 
Considered purchasing a particular M=2.00 M=155 M=2.83 
intervention or program targeted at a SD = 85 SD = .93 SD =.41 
specific student population 
. : . M=1.71 M = 1.79 M =2.14 
Considered scaling up a pilot program SD =.91 SD = 89 SD = .95 
: M = 1.77 M = 2.00 M = 2.40 
Redesigned a program SD =.93 SD = .74 SD = 84 
. ; M = 2.05 M = 2.03 M = 2.50 
Designed professional development SD = 86 SD = .73 SD=.51 
Considered eliminating a program or M=1.75 M = 2.00 M = 2.63 
policy SD = .97 SD =.77 SD =.74 
Considered directing new or additional M=2.29 M=211 M=2.25 
resources (funds and/or people) to a SD= 61 SD = 68 SD = 87 


particular program 
Note: Scale: 0-3. Respondents only answered the questions pertaining to decisions they were involved in. As 
such, the sample size varies by item. 


Conceptual use of research. Conceptual research use was measured with items related to how 
often research expanded leaders’ understanding of an issue, provided a common language or 
framework for thinking about an issue, or changed the way leaders looked at an issue. It was 
highest among district leaders in Aspen Alliance (not shown in table). There, the average rating for 
items on ascale from O to 3 was 1.91 (SD = .51, n = 22). By contrast, the average level of 
conceptual use in Eastern Ridge NIC was 1.53 (SD = .51, n = 28), and in Cypress Design it was 1.56 
(SD = .38, n = 30). 


Trust and synchrony. All three RPPs exhibited high levels of trust, as reported by district leaders. 
In addition, all reported engaging in high levels of trust-building strategies. These include 
respecting one another’s time and making sure that partners left meetings with a clear sense of 
what each partner was doing next. Where RPPs varied—and earned somewhat lower marks from 
district leaders overall—was in the degree to which they achieved synchrony between research 
and practice timelines. Eastern Ridge NIC and Cypress Design district leaders did, however, report 
higher levels of synchrony than Aspen Alliance. Example items in this scale included “The 
researchers and practitioners have different timelines for getting things done” and “The 
researchers’ and practitioners’ schedules are hard to coordinate.” 
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Table 9. Presence of Conditions That Support Partnering 


Partnership 


Aspen 
Eastern Ridge NIC Cypress Design Alliance 

Condition (n= 7) (n = 8) (n= 7) 
Trust M=9.57 M = 10.00 M=9.41 

(Total possible score: 12) SD = 2.30 SD = 1.92 SD = 2.01 

Trust-building strategies M=12.44 M= 12.75 M= 11.43 

(Total possible score: 18) SD =.79 SD = 1.75 SD = 1.62 
Synchrony M = 7.33 M=7.63 M=5.29 
(Total possible score: 12) SD = 1.86 SD=.91 SD =.95 


Note: Scores were determined by adding ratings of individual items for each scale; each item was scored 0-3, 
with 3 indicating higher levels of trust, trust-building activities, and synchrony. 


Table 10 presents a set of correlational analyses that explore the relationships that we 


conjectured would be related to mutual learning within RPPs. We do not seek to identify causal 
relationships but rather to highlight factors that future research might explore. 


Table 10. Correlations of Conditions of RPPs with RPP Influence on Policies and Decisions 


Influence on policy and Consultation in 
Condition practice (r) decisions (r) 
Leader instrumental research use -.04 -.09 
Leader conceptual research use -0.08 -0.03 
Synchrony 44* 22 
Trust 46~ 35~ 
Trust-building strategies 47* 13 


~p<.10.*p<.05. 


Two conditions, synchrony and trust, predicted RPP influence on policy and practice or research 
partners’ involvement in district decision making. Synchrony—the degree to which researchers 
and practitioners were able to “sync up” their work to one another’s needs)—was significantly 
associated with influence on policy and practice. 


Our interviews with RPP leaders point to the complexity associated with achieving synchrony. A 
researcher from Cypress Design described how he has had to negotiate tensions between 
providing timely information in response to district needs and providing validated and reliable 
information. Though the team included students and faculty who were experienced in partnership 
work, this particular researcher was new to it, and the experience raised a number of questions for 
him about communication: 


There’s nothing within this project that prevents you from communicating research 
findings. It’s more just the research endeavor of collect data, analyze data, write a paper, 
send it out for review, wait for—by the time that information gets back to a district, if you 
follow only those formal channels, there’s such a lag. There are informal conversations, 
like, “We’re learning this from analyzing our data. And we have not finished the analysis by 
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any means, so we’re seeing some interesting things. Here’s what we've been seeing.” 
Anything we can do to just get that information out to the district sooner and without that 
lag time, is terrific. We do a pretty good job of that informally, but then, I’m new to this, too, 
because | wonder: Is it right to share some of those findings when they’re not yet formal 
and when they aren’t yet maybe fully validated and reliable? What's the right way? 
Because that’s high-stakes, to share what we’re finding in research. What if we share 
something that isn’t valid or, and reliable yet, and then the district acts on it? Then we go 
on later to find out, well, that actually really wasn’t the case? There’s a tension there. You 
want to get that information out, but there’s a risk in getting it out prematurely. 


The use of trust-building activities within the partnership—such as making an effort to be clear in 
communication and making sure that partners were able to leave meetings with a clear 
understanding of what each partner was going to do next—was associated with RPP influence on 
policy and practice. A district leader from Cypress Design described how trust within the RPP 
developed when both the district and research partner were able to offer their own perspectives 
and opinions and independently take up responsibilities related to partnership work: 


Our trust and our ways of working together really mirrored what we want the teacher 
leaders to do with their department. The more we built trust, and trusted each other, and 
gave each other voice and responsibilities, and said, “Okay, it’s yours. | trust you. 
Whenever you're ready with it, show it to me.” We ourselves experienced that whole 
cycle....All those little things build trust, and then that’s when they can really do the work. 


Asense of trust in the partners was also related to influence and consultation in decision making, 
but the correlation was not statistically significant. The interviews do, however, provide evidence 
to support the relationship between trust and consultation. For example, a district leader in Aspen 
Alliance noted that she perceived the trust between the district and the RPP as one of the most 
beneficial parts of the partnership, and necessary for the collaboration: “It requires a lot of—| 
mean | think, like all of this, this work. When you're really, really collaborating so much, it takes a 
lot of trust. A lot of assuming we’re all in this together.” 
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Spread of Research-Based Ideas from RPPs to District Leaders 


Educational leaders’ engagement with partners can influence the collective knowledge of the 
district central office, which is another dimension of organizational learning.*? Below, we examine 
how diffused particular research-based ideas from RPP partners were within these organizations. 


Key findings 
e Inallthree partnerships, we saw evidence of the key research-based ideas from the 
RPP being taken up in district leaders’ interviews. 


e InEastern Ridge NIC, the most common idea about mathematics—mentioned by 50% 
or more of the interviewees—was about growth mindset; the most commonly cited 
ideas from improvement science were about NICs, disciplined inquiry, and broad ideas 
about improvement science. 


e InCypress Design, ideas named by more than half of interviewed leaders included 
those related to the professional development model, including the need for learning in 
professional communities, the role for facilitators, attending to facilitators’ learning 
needs, and the use of video clips. 


e InAspen Alliance, more than half of district leaders made reference to variation across 
schools in professional development uptake and instructional practices. Just under half 
referenced the differential in supports available for science as compared with 
mathematics and the importance of instructional leadership. 


e Additional analysis shows variation by site in the degree to which the big ideas were 
congruent with research partners’ meanings and in whether leaders attributed the 
ideas to the external partners in their discussions. 


To examine the spread of ideas introduced by research partners into partner districts, we 
conducted a “big ideas” analysis. First, we worked from interviews, existing publications, and 
member checks to identify and define a set of research-based “big ideas” deemed to be central to 
the core work of each partnership. Next, we identified a set of key words and phrases associated 
with each big idea and searched for these terms across the corpus of district leader interview data 
from each site. This stage of the analysis produced a set of findings regarding the salience of each 
big idea among district leaders. Finally, we read through each excerpt that we had identified 
through the keyword search, and coded these excerpts for congruence and attribution. 


Congruence refers to the extent to which a district leader exhibited an understanding of the big 
idea that was consistent with the research partner’s definitions. Attribution to the external 
partner refers to indications within the excerpt that the big idea is associated with partnership 
work in some way. Below we present results of the analysis for each RPP. We do not provide 
citations for these big ideas, as doing so would reveal the identities of some of the research 
partners. 
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Uptake of Research-Based Ideas in Eastern Ridge NIC 


Two sets of big ideas were introduced by researchers in Eastern Ridge NIC, one set related to 
mathematics learning and a second set related to improvement science. Specifically, there were 
four big ideas about mathematics learning in circulation: 


e Growth mindset in mathematics, or the idea that students believe they can learn anything 
and can grow their intelligence. 


e Multiple ways of engaging in mathematics—that is, different ways students can 
successfully approach mathematical practices. 


e Theimportance of depth, not speed in problem solving—that is, teachers should support 
students to go in depth in their mathematics problem solving, rather than emphasize 
speed. 


e Using the rich task routine, developed in the RPP, for presenting students with and 
engaging them in mathematical tasks that are challenging but accessible. 


And there were five big research-based ideas related to improvement science circulating in 
Eastern Ridge NIC: 


e The broad idea of improvement science as an avenue for accelerating improvement in 
schools in a way that is informed by data and evidence. 


e Disciplined inquiry, particularly through the use of rapid, regular, iterative testing using 
Plan-Do-Study-Act cycles in order to learn, fail, and improve quickly. 


e The potential of accelerating improvement through NICs—that is, networks of teams that 
are engaged in coordinated efforts to make progress toward a specific aim. 


e Aworking theory of improvement, represented often in the form of a driver diagram, 
which specifies the strategies that the network conjectures it will use to accomplish its 
aim, and the logic behind those strategies. 


e The idea of measurement as fundamental to monitoring progress toward aims. 


Among practice participants in Eastern Ridge NIC (n = 28), the most commonly cited idea about 
mathematics was that of a growth mindset in mathematics (mentioned by 93% of interviewees). 
The most commonly cited ideas from improvement science related to networked improvement 
communities (89%) and disciplined inquiry (86%). As Table 11 below shows, only one other idea— 
improvement science more broadly—was cited by 50% or more of the interviewees. 


Technical Report No. 4 e 
National Center for Research in Policy and Practice 


36 


Table 11. Invocation, Congruence, and Attribution of Big Ideas in Eastern Ridge NIC 


% of mentions 
congruent with %ofmentions 


external attributed to 

% of leaders partner external 
Big idea invoking big idea meaning* partner 
Growth mindset 93% 22% 38% 
Networked communities 89% 50% 10% 
Disciplined inquiry 86% 22% 18% 
Improvement science 61% 23% 32% 
Measurement 46% 46% 19% 
Depth, not speed 39% 73% 7% 
Theory of improvement 39% 82% 29% 
Rich tasks 36% 45% 30% 
Multiple ways of engaging in 14% 67% 6% 


mathematics 
*Remaining mentions were coded either non-congruent to external partner meaning or ambiguous. 


It was sometimes difficult to tell whether the use of a big idea was congruent with its 
representation in research, in part because the mention was brief. This was particularly so for two 
of the ideas currently in wide circulation in education and mathematics, namely the idea of growth 
mindset (22%) and rich tasks (45%). By contrast, nearly three quarters (73%) of mentions of 
“depth, not speed” were congruent with external partners’ meanings, and two thirds (67%) of 
mentions of the idea of multiple ways of engaging mathematics were consistent with how 
researchers might invoke the idea. 


Use of ideas related to improvement science were more likely to be congruent with use in 
research when the working theory of improvement was invoked (82%). By contrast, it was often 
difficult to tell whether practitioners’ use of the terms “improvement science” or “discipline 
inquiry” were congruent or not. 


Finally, district leaders sometimes attributed these big ideas directly to the external partners who 
were advancing them in their work together. In 38% of the mentions of “growth mindset,” leaders 
explicitly named the mathematics education researcher who is connected with this idea. Similarly, 
in 32% of the mentions of “improvement science,” leaders named the nonprofit involved with the 
partnership that brought ideas about improvement science to the table. 


Uptake of Research-Based Ideas in Cypress Design 


In Cypress Design, there were a total of seven big research-based ideas introduced by external 
research partners, six of which focused on professional development. These six were: 


e Teacher learning within professional communities—that is, the potential of teachers to 
learn from colleagues in a sustained way through discussion of practice. 


e Attending to the teacher role in facilitating learning in the classroom. 
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e Using video clips of teachers’ own practice in order to foster reflection and growth. 


e The idea of cycles of ongoing learning—that is, the need to organize learning in 
communities around cycles of systematic inquiry, including planning, implementing, and 
reflecting on tasks implemented in the classrooms. 


e Theroles for facilitators within professional learning, which were highly specified within 
the model being adapted. 


e Facilitator learning needs—that is, the idea that facilitators themselves need 
opportunities to learn how to engage in their work of facilitating teacher learning. 


The one idea related directly to students was that of engaging them in solving rich problems, an 
idea very similar to “rich tasks” in Eastern Ridge NIC. 


As Table 12 below shows, most salient to the 18 district leaders interviewed were learning in 
professional communities (83%), the role for facilitators (72%), facilitators’ learning needs (56%), 
and using video clips (50%). These were named by half or more of the people interviewed in the 
central office. Less salient to leaders were mentions of solving rich math problems (39%), cycles of 
ongoing learning (39%), or attending to the role of teacher (22%). 


Table 12. Invocation, Congruence, and Attribution of Big Ideas in Cypress Design 


% of mentions 
congruent with %of mentions 


% of leaders external attributed to 

invoking big partner external 
Big idea idea meaning partner 
Learning in professional communities 83% 70% 50% 
Role for facilitators 72% 46% 55% 
Facilitator learning needs 56% 55% 50% 
Using video clips 50% 63% 34% 
Solving rich problems 39% 87% 33% 
Cycles of ongoing learning 39% 47% 47% 
Attending to teacher role 22% 100% 20% 


About half or more (46% and above) of all big ideas were congruent with how researchers framed 
them. Attending to the teacher role (100%), solving rich math problems (87%), and learning in 
professional communities (70%) were the most congruent big ideas. There were also non- 
congruent ways some of these ideas were invoked. The most noticeable was with “using video 
clips,” where 22% were judged non-congruent and 15% were judged ambiguous (not shown). 
Here, several of the non-congruent ways of engaging with video invoked an alternate professional 
development model being used in the district that followed slightly different protocols for using 
video data. 


Finally, district leaders often attributed big ideas they mentioned to the RPP or specific external 
partners. In over half of the instances where district leaders discussed the most salient ideas from 
their partners (learning in professional communities, role for facilitators, and facilitator learning 
needs), they named the external partner, the partnership, or key individuals. 
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Uptake of Research-Based Ideas in Aspen Alliance 


The eight big research-based ideas discussed in Aspen Alliance were interim and past findings 
from the research team’s ongoing analysis work. These findings included: 


e Fewer supports for science than math. 

e Variation in uptake of professional development at the school level. 
e Variation in uptake of instructional shifts at the teacher level. 

e Overall evidence of improvement in student outcomes over time. 


e Specific evidence of improvement for all student subgroups, including those with lower 
incoming achievement. 


e Evidence that teacher professional development matters for student outcomes. 
e Evidence that school-level instructional leadership matters for student outcomes. 


e Evidence that, overall, elementary school teachers were better prepared than high 
school teachers for adopting new standards-aligned instructional practices. 


In addition to tracking ideas central to the work of this specific project, the Aspen research team 
also analyzed two ideas drawn from previous work conducted by the research organization in the 
district, on college and career predictors and school improvement indicators. The interview 
protocol did not include specific questions about earlier projects, though we did invite participants 
who were not familiar with the current project to reflect on any other experiences they had had 
with the research organization. 


As seen in Table 13, more than half of the 17 district leaders included in this analysis made 
reference to variation across schools in professional development uptake and instructional 
practices (56%). Just under one half referenced the differential in supports available for science as 
compared with mathematics (44%) and the importance of instructional leadership (44%). Around 
one third addressed ideas about overall improvement over time (39%) and compared elementary- 
level and high school-level resources and preparation (33%). Fewer than one quarter commented 
on the effectiveness of professional development or variation across teachers. Our keyword 
analysis did not return any excerpts relevant to improvement specifically for low-achieving 
students among district leader interviews. 


Each of the previous projects, related to college and career predictors and to school improvement 
indicators, were referenced by almost half of district leaders (44%). Given that these projects 
were not the focus of our main study, this suggests that that the rates of take-up of ideas 
associated with the specific project are not reflective of the full range of ideas that may be shared 
in this RPP. 
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Table 13. Invocation, Congruence, and Attribution of Big Ideas in Aspen Alliance 


% of mentions % of mentions 


% of leaders congruent with attributed to 

invoking big external partner external 
Big idea idea meaning partner 
School-level variation 56% 82% 6% 
Fewer supports for science than math 44% 77% 23% 
Instructional leadership matters 44% 69% 8% 
Improvement over time 39% 57% 36% 
Elementary schools better prepared 33% 33% 0% 
Professional development matters 22% 50% 50% 
Teacher-level variation 22% 100% 0% 
Improvement for low-achieving 0% 0% 0% 

: StU eS 

College and career predictors 44% 42% 81% 
School improvement indicators 44% 45% 73% 


Overall, the majority of uses of these big ideas were congruent with how researchers conceived of 
them. Notable exceptions to that trend concerned claims about the importance of professional 
development (50%) and evidence that elementary schools were better prepared than high schools 
(33%) for adopting new practices. 


Of the big ideas related to mathematics, one half (50%) of the discussions of professional 
development mattering for improvement efforts were attributed to the external partner’s 
findings. Notably, the student achievement indicators and school improvement indicators—two 
previous projects that were not the central focus of this study—were very often attributed to the 
external partner (in 81% and 73% of cases, respectively), suggesting these earlier RPP projects left 
aclear imprint on the district. 
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Discussion 


The findings of this study contribute to the field’s understandings of RPPs in four key ways: the 
relationship of the RPP’s organization to goals and activities; the research produced within an 
RPP; the kinds of influences that RPPs can have on districts’ policies, practices, and decisions, 
given supportive conditions; and the imprint that ideas from an RPP can have on district leaders’ 
thinking. 


The Relationship of Form and Composition of Organization to Goals and Activities 


In the typology of RPPs that informed the case study sampling,” these partnerships were 
presented as having distinct forms, each with an associated set of goals and activities. Although 
the RPP sites were sampled as paradigmatic cases*? of the three types—RA, DRP, and NIC—our 
findings suggest there were a number of ways that goals and activities overlapped across type. 


Participants in all of the RPPs reported engaging in activities that a prototypical RA was 
conjectured to do, from developing a deeper understanding of a persistent problem of practice to 
setting up a data infrastructure. Thus, it is possible that these RA-like activities are common to 
most RPPs—a finding supported by an earlier study of IES-funded RPPs conducted by NCRPP.** 


At the same time, there was considerable overlap in partnership activities between the two other 
partnership types, NICs and DRPs. For example, we see many instances of design-based research 
projects using tools from improvement science, such as practical measures.” This and other 
overlapping activities may reflect a shared intellectual lineage across the two traditions of 
engagement, including a common focus on iterative change processes.” In the case of Cypress 
Design, the integration occurred because several members of the district leadership team and two 
members of the research team were part of another RPP focused on practical measurement (RPP 
leader, personal communication, 6/28/20). It may also reflect ongoing borrowing and learning 
taking place across disparate partnerships as research products and processes are shared. In this 
way, the boundaries around clear-cut “types” of RPPs have blurred, with hybridity emerging in the 
field. This is a potential strength as the field grows, because it widens the tools RPPs can use to 
address a wider range of questions. 


While some activities were similar, partnerships differed considerably in the composition of teams 
doing this collaborative work. In Eastern Ridge NIC, the research organization mainly focused on 
specifically supporting continuous improvement work—reflecting the specific focus of NICs on 
using improvement science methods. It also went beyond district leaders to include leaders from 
the county education office, which served as the hub. This participation of county-level 
stakeholders was unique to Eastern Ridge NIC, reflecting both the organizational features specific 
to that site as a regional service provider to school districts and typical NIC goals. (It is a 
definitional goal of NICs to have an organizational hub of some kind to bring together a network of 
participants across traditional boundaries of educational organizations.) 


Cypress Design included participation from district leaders, in a range of roles relating to 
supporting mathematics instruction, and from a university-based research team. This reflects the 
typical DRP goal of engaging collaboratively with district leaders responsible for implementing or 
overseeing co-designed programs or practices. Cypress Design also received the support of a 
partnership-brokering organization. This additional participant reflects the specific history and 
capacities of the Cypress Design district and the research team’s university home. 


Technical Report No. 4 « A1 
National Center for Research in Policy and Practice 


Finally, the participation of the long-standing Aspen Alliance team members along with district 
leader partners reflects typical RA goals. However, in addition to the partnership work of Aspen 
Alliance addressed here, we note that a university-based professional development provider was 
also working alongside the practice leaders in the Aspen Alliance district. They were not part of 
the partnership; however, their work was an object of study for the researchers in the partnership. 


Taken together, these findings provide empirical evidence of the need to soften the boundaries of 
different RPP types and to expect there to be—even within paradigmatic cases—examples of goals 
and activities that are broad and encompass problem finding, design and implementation, and 
various forms of research and inquiry. The findings also point to the complexity of RPPs, which 
often encompass multiple kinds of relationships and projects that require different methods to 
address emerging questions. 


Research Products 


All three RPPs produced multiple kinds of research products, reflecting the nature of RPPs as 
knowledge-building organizations. The single most common manner of sharing this knowledge 
was through conference presentations—including to academic, practitioner, and hybrid 
audiences— particularly for Cypress Design and Aspen Alliance. All three partnerships also 
produced technical or analytic reports and peer-reviewed journal articles. In addition, Eastern 
Ridge NIC developed a particularly large number of educational resources that can be shared with 
others. This finding supports other research?’ that suggests that RPPs have commitments to 
sharing findings and ideas with broader communities (i.e., beyond their immediate partners), 
although those audiences and the form of that communication may vary. In this sense, the term 
“research products” may be a misnomer; the products themselves are much more than 
presentations of findings. 


The titles of these different products reflect the respective goals of the partnerships. In one form 
or another, all three indicated knowledge-building contributions relating to improvement in 
mathematics instruction and student outcomes. Eastern Ridge NIC products reflected a 
particularly heavy emphasis on the “network” form of the partnership, and on interventions 
designed to improve mathematics instruction. Cypress Design products indicated a particular 
focus on professional development for mathematics teachers, reflecting the nature of the co- 
designed project. Aspen Alliance products emphasized standards and student achievement in 
mathematics and science. 


The majority of these products were authored by members of research teams, which reflects 
traditional professional expectations in universities and research organizations. However, 
practice partners sometimes served as authors, co-authors, and co-presenters. This is reflective of 
the collaborative nature of RPPs and of the multiple roles and forms of participation that take 
place within them. Co-authorship of research products occurred in both Eastern Ridge NIC and 
Cypress Design, but not Aspen Alliance. This difference in the degree of close collaboration in the 
production of research products aligns with differences in research activities of the respective 
RPPs. For example, both Eastern Ridge NIC and Cypress Design (but not Aspen Alliance) involved 
joint participation of researchers and district leaders in co-design and testing of programs and 
initiatives. 
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Influence of RPPs on District Leaders’ Practices, Policies, Routines, and Decisions 


A key goal of many RPPs is to support improvement in student outcomes.°? The process of 
improving student outcomes, however, often requires changes to policies and practices that are 
created or supported at the district level. Thus, to support improvements to student outcomes, an 
RPP must first influence policies and practices at the district level. Each of the RPPs in this study 
did so, as reported by the practice leaders within that RPP. 


The most common source of influence was the design of professional development in the district. 
Between two thirds and four fifths of leaders said the RPP’s activities influenced district practice 
in this area, depending on the partnership. On the one hand, this is a logical area for influence, in 
that each of the projects was related to implementation of significant professional development 
for teachers and teacher leaders. But it is still noteworthy that participants in each RPP 
characterized the partnership as having influenced the design of professional development, since 
this is an area that can have a direct impact on teaching practice and, subsequently, student 
outcomes. 


Equally significant was that there were other areas of influence beyond the focal project, such as 
introducing new frameworks or sets of ideas to help think about the goals and how to address the 
focal problems of districts. In addition, in all three RPPs, practitioners said that practices that had 
been developed or modeled in the partnership had been integrated into the district. This suggests 
that an RPP can have influence beyond a narrow project focus in ways that support district goals. 
There were some notable differences across partnerships: More practice partners in the two 
design-focused RPPs (Eastern Ridge NIC and Cypress Design) said their research partners had 
influenced the design and integration of programs and practices in the district. 


The research partners were consulted in district decision making as well, according to practice 
leaders. The most common type of decision for which partners were consulted was the design of 
professional development. Again, this is logical, given the focus of joint work in all three 
partnerships. There were additional areas of consultation that went beyond the focus of 
immediate projects, such as when considering scaling up a pilot program, redesigning a program, 
or redirecting resources toward particular programs. These findings may speak to the 
“consultative” role that RPPs can play to district leaders,°*° as researchers bring expertise and can 
broker connections to research beyond the immediate project. 


The degree to which participants reported that RPPs influenced district policy and decisions was 
associated strongly with both the degree of synchrony reported and with trust. Synchrony refers 
to the degree to which researchers and practitioners can align their activities in time, so that the 
research is carried out in a way that is timely and fits within decision making cycles. Trust-building 
activities were also linked to more influence. These two findings confirm what many previous 
studies have found with respect to RPPs,°° and what RPP leaders themselves assert*! are 
important ingredients for a successful partnership. 


Spread of Research-Based Ideas from the Research Partners to District Leaders 


In addition to building knowledge broadly and contributing to changes in district practices and 
policies, RPPs are often seen as holding particular promise for facilitating the spread of research- 
based ideas into sites of practice. We found evidence that the research-based ideas introduced by 
all three RPPs did indeed spread to district leaders’ thinking about mathematics teaching and 
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learning. In each partnership, several research-based ideas were named by half or more of 
interviewed educational leaders as key contributions by the research team 


Not all references to research-based ideas were congruent with the ways researchers might 
interpret and use those ideas, although brief mentions of them made it hard to judge congruence 
in many cases. In addition, when district leaders referenced big ideas, they sometimes named the 
external partner or RPP, making us more confident that the RPP contributed to the imprint of this 
idea. Yet, at other times, leaders made mention of such ideas—reflecting their salience in district 
work—but did not explicitly connect them to external partners. Perhaps counterintuitively, the 
finding that district leaders do not always attribute research-based ideas directly to external 
partners is consistent with the collaborative nature of RPPs. The shared responsibility across 
researcher teams and district leaders for identifying partnership goals means that district leaders 
are able to select partners and projects that reflect existing district priorities. Moreover, as key 
ideas from external partners are integrated into district policies and practices, district leaders may 
feel a greater sense of ownership of these ideas and refer to them in contexts outside of discussion 
of the partnership itself. 


Conclusions 


RPPs can have an influence on policies and practices in partner districts, and this can affect what 
goes on inclassrooms. They can influence the design of professional development that reaches 
teachers, potentially changing how they teach and how students learn in their classrooms. 
Moreover, the influence of RPPs can extend beyond their immediate work. These partnerships 
can affect the thinking, practices, and decisions of teachers and teacher leaders, as well as district 
leaders who are familiar with RPP work but not directly involved in it. 


Although we do not know what impact, if any, these RPPs have had on student outcomes directly, 
the fact that they have influenced key levers for improving outcomes is significant. Other studies 
point to the capabilities of RPPs to design interventions that can powerfully impact teacher and 
student learning.” Still, more studies are needed to investigate empirically and causally the links 
among RPP activities, changes to practice, and improvements in student outcomes. 


Finally, this study confirms that although RPPs may have some common elements—collaborative 
relationships between researchers and practice, analysis of implementation and outcomes of 
policies and practices in partner districts, and engagement in joint sensemaking of data—they can 
and do differ in important ways. Specifically, RPPs vary in the degree to which they engage in co- 
design and make use of tools, such as the tools of improvement science. Simply put, there is no 
single model for an RPP, and different ways of organizing have their own affordances and 
constraints. 
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